1. Field of the Invention
The present invention relates to image processing apparatuses and methods, and media therefor, and in particular, to an image processing apparatus and method that easily implements processing such as three-dimensional editing on a two-dimensionally displayed three-dimensional object, and a medium therefor.
2. Description of the Related Art
Various methods have been proposed that implement various processes on a two-dimensional image and extract information necessary for the processes from the two-dimensional image. Documents describing the methods include James D. Foley, Andries van Dam, Steven K. Feiner, and John F. Hughes, xe2x80x9cComputer Graphics, principles and practicexe2x80x9d, ADDISON-WESLEY PUBLISHING COMPANY, 1996 (hereinafter referred to as xe2x80x9cDocument 1xe2x80x9d), Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik, xe2x80x9cModeling and Rendering Architecture from Photographs: A hybrid geometry-and-image-based approachxe2x80x9d, proceedings of SIGGRAPH 96, pp. 11-20 (hereinafter referred to as xe2x80x9cDocument 2xe2x80x9d), Oliver Faugeras, xe2x80x9cThree-dimensional computer versionxe2x80x9d, The MIT press (hereinafter referred to as xe2x80x9cDocument 3xe2x80x9d), Kenneth P. Fishin, and Brian A. Barsky, xe2x80x9cFamily of New Algorithms for Soft Fillingxe2x80x9d, proceedings of SIGGRAPH 84, pp. 235-244 (hereinafter referred to as xe2x80x9cDocument 4xe2x80x9d), Pat Hanrahan and Paul Haeberli, xe2x80x9cDirect WYSIWYG Painting and Texuturing on 3D Shapesxe2x80x9d, proceedings of SIGGRAPH 90, pp. 215-233 (hereinafter referred to as xe2x80x9cDocument 5xe2x80x9d), Youichi Horry, Ken-ichi Anjyo, and Kiyoshi Arai, xe2x80x9cTour Into the Picture: Using a Spidery Mesh Interface to Make Animation from a Single Imagexe2x80x9d, proceedings of SIGGRAPH 97, pp. 225-232 (hereinafter referred to as xe2x80x9cDocument 6xe2x80x9d), and Michael Gleicher, xe2x80x9cImage Snappingxe2x80x9d, proceedings of SIGGRAPH 95, pp. 183-190 (hereinafter referred to as xe2x80x9cDocument 7xe2x80x9d).
In Document 1, image processing called xe2x80x9ctwo-dimensional paintxe2x80x9d is described in which a computer is used to perform processing in the same way as a designer or the like draws a picture on paper using a paintbrush or airbrushing (a technique that draws a picture by spraying paints on paper).
In this type of conventional two-dimensional paint, even when a three-dimensional object is displayed in an image, the image itself is treated on a two-dimensional plane. Accordingly, when characters are rendered irrespective of the direction of the three-dimensional object in a three-dimensional space which is displayed on the two-dimensional plane, or a figure is added, the image looks unnatural.
For example, in the case where a house-shaped three-dimensional object, as shown in FIG. 1A, is displayed, and characters are rendered on walls of the house without the direction of the characters, the characters do not look as if they are written on the walls, as shown in FIG. 1B. For adding a parallelepiped room to a wall of the house, in the case where a rectangle is rendered on the wall without the direction of the rectangle, the image looks unnatural, as shown in FIG. 1C. In the case where a cylinder is displayed in a two-dimensional image as shown in FIG. 2A, when characters are rendered on the side surface without ignoring the curvature of the side surface, the characters do not look as if they are written on the side surface, as shown in FIG. 2B.
Accordingly, in order that an image may not look unnatural in two-dimensional paint, it is required that a character or figure be rendered being transformed so as to match the direction of a three-dimensional object displayed in the two-dimensional image. Performing operations for the rendering requires a degree of experience.
Therefore, there is a method in which a user uses a ten-key pad or graphical user interface (GUI) to input an angle of inclination of the three-dimensional object displayed on the two-dimensional image so that a computer uses the input to transform a new character or figure to be rendered. In this method, the user must adjust the angle of inclination of the three-dimensional object to be input to the computer while viewing the rendered character or figure so that the rendered character or figure does not look unnatural. The adjustment also requires a degree of experience.
As described above, when the user instructs the rendering of a character or figure ignoring the direction of a three-dimensional object displayed on a two-dimensional image, the computer cannot display the character or figure as it looks natural, in other words, an image looking as if the character or figure was originally positioned cannot be obtained. This is due to lack of information on the position of the three-dimensional object displayed in the two-dimensional image and information (a position at which a landscape or the like was observed in the case where the two-dimensional image was obtained by performing image capture on a picture) on the position of image capture for the two-dimensional image.
Accordingly, there is a method that uses computer vision to find, from a two-dimensional image, the position of a three-dimensional object in a three-dimensional space displayed in the two-dimensional image, and the image capture position of the three-dimensional object.
In other words, in Document 2, a method is disclosed in which a three-dimensional shape such as a parallelepiped is correlated using a GUI with a structure in a plurality of photographs, and the size of the structure and the photographing position are found. In Document 3, other various methods for finding the position of a three-dimensional object and an image capture position on the object are disclosed.
In the above-described methods using the computer vision to compute a three-dimensional object position and an image capture position on the object, the principles of triangulation are used. This requires a plurality of images obtained by performing image capture on the same three-dimensional object from plurality of image capture positions. However, when two-dimensional paint is performed, the images are not always prepared, and when the two-dimensional images are photographs of a picture, the plurality of images is not used in general.
If the plurality of images obtained by performing image capture on the same three-dimensional object from the plurality of image capture positions can be prepared, the corresponding positions (e.g., vertices of the roof, etc., of a structure as a three-dimensional object) of the same three-dimensional object displayed on the images must be designated in order for the computer to compute the position of the three-dimensional object and the image capture position. The operation of designation for the images is complicated, and requires a time. In addition, in the case where based on the position (in the three-dimensional space) of the three-dimensional object displayed in the two-dimensional image and the image capture position, three-dimensionally natural rendering is performed, three-dimensional data, such as three-dimensional coordinates generated when the three-dimensional object is viewed from the image capture position, must be processed, which requires a great amount of operations for the processing.
In Document 4, a method for color processing for two-dimensional paint is disclosed.
In two-dimensional paint, the user uses a GUI to select a color for rendering and to perform rendering using the selected color. The color and brightness of a three-dimensional object displayed in the two-dimensional image vary depending on the positional relationship between the direction of the object surface and illumination. For example, painting an entire surface of the three-dimensional object in the same color (the same RGB levels) causes an unnatural rendering result. Accordingly, for obtaining a natural rendering result, a painting color must be gradationally changed considering the positional relationship between the direction of a surface to be painted and illumination. In particular, in the case where a surface to be painted is curved, pixel levels needs to be sequentially changed so that the rendering result is natural. Thus, a painting color must be selected for, for example, several pixels, which requires a great amount of operations.
Accordingly, in Document 4, a method called xe2x80x9ctint fillxe2x80x9d is disclosed in which in the case where a pixel color is a mixture of two colors and the two colors are known, either color can be changed into another color. The use of tint fill can efficiently cope with an area in which a foreground and a background are mixed, but cannot cope with object shading and light reflected by the object.
In Document 5, a method that transforms and pastes a character or handwritten figure input by operating a tablet or mouse along the surface of a three-dimensional object displayed in a two-dimensional image is disclosed.
Since the method disclosed in Document 5 uses three-dimensional information (e.g., the position of the three-dimensional object in a three-dimensional space, an image capture position, size, inclination, etc.) on the three-dimensional object displayed in the two-dimensional image, the method in Document 5 needs to recognize the three-dimensional information by using computer vision. However, when computer vision is used to recognize the three-dimensional information, a plurality of images are required as described above, and an amount of operations greatly increases.
Although the method in Document 5 can also use a three-dimensional model of a three-dimensional object generated in the computer, it must process three-dimensional data and requires a great amount of operations.
Regarding three-dimensional model generating, Document 2 discloses a technique that includes the steps of using computer vision technology to extract an outline of the shape of a three-dimensional object displayed in the two-dimensional image from the two-dimensional image, pasting a captured image onto the shape surface (performing texture mapping), and using these steps to generate a two-dimensional image viewed from a free viewpoint (captured from a free image capture position). Similar to the above-described cases, this technique requires a plurality of images because it uses computer vision technology to extract, from a two-dimensional image, an outline of the shape of a three-dimensional object displayed in the two-dimensional image.
In Document 6, a method is disclosed in which based on a single two-dimensional image, an image viewed from a viewpoint excluding the image capture position of the two-dimensional image is generated. This method only generates an image viewed from a viewpoint excluding the image capture position, and it is based on the assumption that a three-dimensional object displayed in a two-dimensional image or another adjacent object has a very simple shape of a plate. Accordingly, almost no serious problem occurs when an image viewed from another viewpoint is approximately generated. However, when a character or figure is added to the two-dimensional image, it is impossible to make the rendering result natural. For example, in the case where a cylinder is displayed in a two-dimensional image, a natural image formed by adding a character or figure on the side surface of the cylinder cannot be obtained.
In addition, it is required in many cases that when a two-dimensional image is processed, the user designates predetermined characteristic points (e.g., vertices) of a three-dimensional object in the two-dimensional image. The characteristic points are designated by operating a mouse or input pen, but it is difficult to accurately designate vertices of the three-dimensional object in the two-dimensional image. Accordingly, Document 7 discloses a method that includes the steps of extracting an edge (a portion whose brightness and color greatly change) in the two-dimensional image, and moving a cursor on the edge when the cursor is in the vicinity of the edge. Since all edges in the two-dimensional image are detected in the method in Document 7, sensitive response to noise or a texture in the two-dimensional image occurs, and the cursor may be moved to an unnecessary position.
Although many type of conventional image processing technology have been proposed as described above, a method particularly adapted for editing or processing on a two-dimensional image, such as three-dimensional addition and deletion of a character or figure, movement, transformation, color or pattern change, and illumination change, has not been proposed yet.
According to the above-described conventional methods, three-dimensional editing or processing a degree of experience of operation and a time, or a plurality of images in which the image of the same object is captured from different directions. An amount of computation extremely increases, and a cursor may be unnecessarily moved. In addition, in order to delete a three-dimensional object or the like in a two-dimensional image, it is required that in a region of deletion, the background of the two-dimensional image be reproduced. However, it is difficult for the above-described methods in Documents 1 to 7 to reproduce the background.
Accordingly, it is an object of the present invention to provide an image processing apparatus and method for easily implementing processing such as three-dimensional editing on a two-dimensional image, and a medium used for the image processing apparatus and method.
To this end, according to an aspect of the present invention, the foregoing object is achieved through provision of an image processing apparatus for generating, from a two-dimensional image, expanded images obtained by expanding, on a two-dimensional plane, a plurality of surfaces constituting a three-dimensional object displayed in the two-dimensional image. The image processing apparatus includes an operating unit for designating shape information on the shape of a surface among the surfaces constituting the three-dimensional object, and characteristic points in the two-dimensional image which are included in the surface, a computing unit for computing, based on the shape information and the characteristic points, a transform expression for transforming each surfaces constituting the three-dimensional object into each expanded image, a transforming unit for transforming, based on the transform expression, each of the surfaces constituting the three-dimensional object displayed in the two-dimensional image into one of the expanded images, and a connecting unit for generating a connected image by connecting the expanded images of the surfaces in accordance with connecting relationship among the surfaces.
According to another aspect of the present invention, the foregoing object is achieved through provision of an image processing method for generating, from a two-dimensional image, expanded images obtained by expanding, on a two-dimensional plane, a plurality of surfaces constituting a three-dimensional object displayed in the two-dimensional image. The image processing method includes a computing step for computing, based on shape information on the shape of a surface among the surfaces constituting the three-dimensional object and on characteristic points in the two-dimensional image which are included in the surface, a transform expression for transforming each of the surfaces constituting the three-dimensional object into each of the expanded images at the time of designation of the shape information and the characteristic points, a transforming step for transforming, based on the transform expression, each of the surfaces constituting the three-dimensional object displayed in the two-dimensional image into one of the expanded images, and a connecting step for generating a connected image by connecting the expanded images of the surfaces in accordance with connecting relationship among the surfaces.
According to a further aspect of the present invention, the foregoing object is achieved through provision of a computer-readable medium for causing a computer to execute an image-processing computer program for generating, from a two-dimensional image, expanded images obtained by expanding, on a two-dimensional plane, a plurality of surfaces constituting a three-dimensional object displayed in the two-dimensional image. The program includes a computing step for computing, based on shape information on the shape of a surface among the surfaces constituting the three-dimensional object and on characteristic points in the two-dimensional image which are included in the surface, a transform expression for transforming each of the surfaces constituting the three-dimensional object into one of the expanded images at the time of designation of the shape information and the characteristic points, a transforming step for transforming, based on the transform expression, each of the surfaces constituting the three-dimensional object displayed in the two-dimensional image into one of the expanded images, and a connecting step for generating a connected image by connecting the expanded images of the surfaces in accordance with connecting relationship among the surfaces.
According to a still further aspect of the present invention, the foregoing object is achieved through provision of an image processing apparatus for processing an image displaying a three-dimensional object. The image processing apparatus includes a first transforming unit for transforming each of projected images which is displayed in the image to be processed, the projected images obtained by projecting a plurality of surfaces constituting the three-dimensional object, into each of expanded images obtained by expanding the surfaces on a two-dimensional plane, a connecting unit for generating a connected image obtained by connecting the surfaces in accordance with connecting relationship among the surfaces, a processing unit for processing the connected image, a second transforming unit for transforming the expanded image of each surface constituting the processed connected image into the projected image of each surface, and a pasting unit for pasting one of the projected images on the image to be processed.
According to yet another aspect of the present invention, the foregoing object is achieved through provision of an image processing method of an image processing apparatus for processing an image displaying a three-dimensional object. The image processing method includes a first transforming step for transforming each of projected images which is displayed in the image to be processed, the projected images obtained by projecting a plurality of surfaces constituting the three-dimensional object, into each of expanded images obtained by expanding the surfaces on a two-dimensional plane, a connecting step for generating a connected image obtained by connecting the surfaces in accordance with connecting relationship among the surfaces, a processing step for processing the connected image, a second transforming step for transforming the expanded image of each surface constituting the processed connected image into the projected image of each surface, and a pasting step for pasting one of the projected images on the image to be processed.
According to a more aspect of the present invention, the foregoing object is achieved through provision of a computer-readable medium for causing a computer to execute an image-processing computer program for processing an image displaying a three-dimensional object. The program includes a first transforming step for transforming each of projected images which is displayed in the image to be processed, the projected images obtained by projecting a plurality of surfaces constituting the three-dimensional object, into each of expanded images obtained by expanding the surfaces on a two-dimensional plane, a connecting step for generating a connected image obtained by connecting the surfaces in accordance with connecting relationship among the surfaces, a processing step for processing the connected image, a second transforming step for transforming the expanded image of each surface constituting the processed connected image into the projected image of each surface, and a pasting step for pasting one of the projected images on the image to be processed.
According to the present invention, expanded images of a plurality of surfaces constituting a three-dimensional object in a two-dimensional image can be easily obtained.
According to the present invention, a plurality of surfaces of a three-dimensional object, displayed in an image, can be easily edited.